In some applications such as monitoring and tracking of video and audio content distribution (e.g., broadcast, Internet, etc.), it is desirable to identify different parts of the media with a fine granularity. Granularity refers to the smallest unit of time (or portion) of the media signal which can be reliably identified. For example, this might be a particular point in a TV show, advertisement, movie or song.
Consider a video signal with an embedded watermark. Assume that the same watermark payload is repeatedly embedded in each frame of the video. Under noisy conditions (compression, D/A/D conversions etc.), the watermark detection process can aggregate the watermark signal across several frames since the payload is identical. Aggregation improves the signal-to-noise ratio and provides improved robustness. However, in this example, the watermark signal does not provide the ability to distinguish between different portions of the video signal.
Now consider a video signal with a unique watermark payload embedded in each frame of the video. In this case, the watermark signal provides fine granularity—the ability to identify each individual frame of the video signal. However, under noisy conditions, the watermark robustness would drop since the unique payload does not necessarily allow aggregation of the watermark signal.
A similar issue exists in fingerprinting systems where granularity is achieved by extracting a unique fingerprint for each portion of the media signal. The finer the granularity, the larger the number of fingerprints and larger is the size of the fingerprint database. Increasing the size of the fingerprint database increases the computational cost (and system cost) of the fingerprint search and matching process.
Watermarks provide the ability to serialize media content, i.e., identical copies of the same media signal can be embedded with distinct watermark payloads, whereas fingerprinting cannot distinguish between identical copies. Watermarking involves introducing changes to the media signal and raises the question of perceptibility of the watermark signal. On the other hand, fingerprinting does not involve any change to the media signal.
A combination of watermarking and fingerprinting can address the issues of granularity, robustness and perceptibility, and can allow greater latitude in the design of content identification systems. Combination approaches that take advantage of the complementary strengths of watermarking and fingerprinting are described below.
Combinations of watermarks and fingerprints for content identification and related applications are described in assignees U.S. Patent Publication 20060031684, which is hereby incorporated by reference. Watermarking, fingerprinting and content recognition technologies are also described in assignee's U.S. Patent Publication 20060280246 and U.S. Pat. Nos. 6,122,403, 7,289,643 and 6,614,914, which are hereby incorporated by reference.
Additional examples of audio and/or video recognition are described in U.S. Pat. Nos. 7,174,293, 7,346,512, 6,990,453 and U.S. Patent Publication 20020178410, which are hereby incorporated by reference. For the purposes of this disclosure, these patent documents provide a description of fingerprint technologies that can be combined with watermarking technologies as explained further below.
For additional examples of video recognition techniques. See, e.g., Bhat, D. N. and Nayar, S. K., “Ordinal measures for image correspondence,” IEEE Trans. Pattern Ana. Mach. Intell., vol. 20, no. 4, pp. 415-423, April 1998. Mohan, R., “Video sequence matching,” Proc. Int. Conf. Acoust., Speech and Signal Processing (ICASSP), vol. 6, pp. 3697-3700, January 1998. Oostveen, J., Kalker, T. and Haitsma, J., “Feature extraction and a database strategy for video fingerprinting,” Proc. 5th Int. Conf. Recent Advance in Visual Information Systems, pp. 117-128, 2002. Kim, C. and Vasudev B., “Spatiotemporal sequence matching for efficient video copy detection,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 1, pp. 127-132, January 2005. Lu, J., “Video fingerprinting for copy identification: from research to industry applications”, Proceedings of SPIE, Media Forensics and Security, Vol. 7254, February 2009.